Every year increasing amounts of data are required to be stored and transmitted for various purposes, including business practices and compliance with various laws. The media upon which this data is recorded has a price in dollars to acquire, a price to manage in manpower time, and a price to provide infrastructure such as power and cooling and/or other factors. It is desirable to reduce the cost of all of these factors. It is generally accepted that the cost to manage and provide such infrastructure is a multiple of the cost to acquire the storage media. By reducing the amount of media, other infrastructure costs may be further reduced. The present invention provides means by which data storage and related media may be conserved, recycled, or reused, thereby reducing the total cost of ownership for data storage.
It has previously been demonstrated that it is possible to build a storage subsystem in which all of the physical storage is initially allocated to a pool, examples of which are discussed in the co-pending U.S. patent application, Ser. No. 10/918,329, filed Aug. 13, 2004, entitled “Virtual Disk Drive System and Method.” That pool may then be allocated on demand to other entities accessible by a computing entity for the use of that entity for data storage. The allocation of storage from the pool to the computing entity in the field of the present invention is commonly referred to as “thin provisioning.” This method of only allocating storage on demand uses the implication that storage is in use by the computing entity in that if the computing entity writes data, it intends to store that data for later retrieval. By only allocating storage identified by those specific operations, a significant amount of storage that is not and may never be in use by a traditional storage subsystem may be omitted from the system as a whole, thereby reducing the cost to acquire, maintain, etc.
However, in standard protocols, it is not possible for a computing entity to communicate to a storage subsystem that a specific area which has previously had data stored to it is no longer in use and can now be reused or otherwise released. This data space may have been used for temporary storage, or may simply be no longer valuable enough to retain for further use. Since there is no means available to identify the area that is no longer in use from the perspective of the storage subsystem alone, the data space continues to be maintained by the storage subsystem. In other words, there is no implicit means that can logically exist by which to determine with absolute certainty without examining the data itself to free the previously implicitly allocated storage. Also, it is computationally intensive for the storage subsystem to examine the contents of all data stored by the computing entity. Therefore, the storage system suffers a very severe performance impact, while trying to keep abreast of technology changes in operating or file systems and all possible applications which may use a storage subsystem.
Overall, it is desirable to know exactly which blocks are in use, and which are not for any operating system, and any type of file system to help make thin provisioning as efficient as possible. No standard exists for a user of block storage to indicate to the storage unit that a block is “not in use.” For traditional storage devices, this information has been entirely irrelevant because one physical block is mapped to each addressable block on the storage device by physical representation. In nearly all storage systems incorporating more than one disk device, any given addressable block may in fact be mapped to nearly any (and sometimes more than one) physical block on one or more physical disk devices. With a fully virtualized, thin provisioned storage system, the only information about what blocks are in use is collected implicitly—if a block is written to, it is assumed to be in use. This is an inherently safe assumption. In the case of thin provisioning, physical blocks are allocated to map to user addressable blocks on an as needed basis, based on the user writing to a given addressable block. “Reads” from blocks that have never been written may return dummy data, usually data composed of all zeros, and of the total length requested. The only method in which a block may be released for reuse in this embodiment is if a PITC is made, and the given logical addressable block is again written to, and the previous PITC expires. This, again, implicitly indicates that the block previously allocated is no longer necessary for the integrity of the addressable storage and can be reallocated as necessary, potentially to other volumes.
Certain conditions can cause an abundance of unused addressable blocks in any FS. An extreme example of this would be creating a single very large file that encompasses nearly the entire volume, then deleting the file. The storage subsystem will implicitly allocate the storage required for every write made by the file system, in this case, those to encompass the entire volume. After the file is deleted, the vast majority of the space allocated by the storage subsystem is not needed, but the storage space cannot be released implicitly, thereby consuming resources. Over time, small allocations and de-allocations at the application or file system level can lead to the same result.
Thus, the existing thin provisioning process in a data storage system is hampered by file system operations of an operating system. These file systems do not reallocate freed space, but rather allocate previously unused space to new file writes, i.e. new file writing operations. This method of operation results in large amount of space that have been previously written to in a given partition that are no longer actually storing usable data for the file system. Because the data storage system has no way of knowing what logic block addresses (“LBAs”) are no longer in use by the file system layered upon the block storage provided by the data storage system, over time these now unused blocks will accumulate. This accumulation will eventually require that every point in time copy (“PITC”) taken will refer to previous pages in a page pool, even though that storage is not actually in use.
Because more and more pages are declared “in use” that are not actually in use, operations such as copies, replication and other data movement will take a greater amount of time, and more storage space will be consumed, potentially at all tiers, thereby defeating many space advantages of thin provisioning. One example is that a 1 gb file is written and a corresponding new volume is allocated, and then the 1 gb file is deleted. In the storage subsystem, 1 gb of pages are still allocated in the active PITC and will be carried into the next PITC and so on. Pages may be replaced in later PITCs, however, in the existing system, there is no way of releasing pages that the file system has declared no longer in use. The result is if this supposedly empty volume is copied using internal tools, 1 gb of pages will be consumed in the new copy, even though the volume is empty.
Accordingly, a process of determining when implicitly allocated storage is no longer in use by a computing entity and can be freed for other uses, is desired.